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Abstract 


Estimating the error rates of classifiers or regression models is a fundamental task in machine 
learning which has thus far been studied exclusively using supervised learning techniques. We 
propose a novel unsupervised framework for estimating these error rates using only unlabeled data 
and mild assumptions. We prove consistency results for the framework and demonstrate its practical 
applicability on both synthetic and real world data. 
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1. Introduction 


A common task in machine learning is predicting a response variable y € Y based on an explanatory 
variable x € X. Assuming a joint distribution p(x,y) and a loss function L(y,), a predictor f : X > 
Y is characterized by an expected loss or risk function 


R(f) — E pay LOFE 


For example, in classification we may have X = R4, Y = {1,...,/}, and L(y, f) = I(y 4 f) where 
I(A) = 1 if A is true and 0 otherwise. The resulting risk is known as the 0-1 risk or simply the 
classification error rate 

R(f) = P(f predicts the wrong class). 


In regression we may have X = Y = R, and L(y,f) = (y—¥)*. The resulting risk is the mean 
squared error 


R(f) = E py) — f(x)’. 


We consider the case where we are provided with k predictors f; : X => Y,i=1,...,k (k > 1) 
whose risks are unknown. The main task we are faced with is estimating the risks R(fi),...,R(fx) 
without using any labeled data whatsoever. The estimation of R(f;) is rather based on an estimator 
R(f;) that uses unlabeled data x“),...,x() © p(x). 
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A secondary task that we consider is obtaining effective schemes for combining k predictors 
fi,- --, fk in a completely unsupervised manner. We refer to these two tasks of risk estimation and 
predictor combination as unsupervised-supervised learning since they refer to unsupervised analysis 
of supervised prediction models. 

It may seem surprising that unsupervised risk estimation is possible at all. After all in the 
absence of labels there is no ground truth that guides us in estimating the risks. However, as we 
show in this paper, if the marginal p(y) is known it is possible in some cases to obtain a consistent 
estimator for the risks using only unlabeled data, that is, 


lim R(fi sx... x)= R(f;) with probability 1, i=1,...,k. 
In addition to demonstrating consistency, we explore the asymptotic variance of the risk estimators 
and how it is impacted by changes in n (amount of unlabeled data), k (number of predictors), and 
R(fi),---;R(fz) (risks). We also demonstrate that the proposed estimation technique works well in 
practice on both synthetic and real world data. 

The assumption that p(y) is known seems restrictive, but there are plenty of cases where it 
holds. Examples include medical diagnosis (p(y) is the well known marginal disease frequency), 
handwriting recognition/OCR (p(y) is the easily computable marginal frequencies of different En- 
glish letters), regression model for life expectancy (p(y) is the well known marginal life expectancy 
tables). In these and other examples p(y) is obtained from extremely accurate histograms. 

There are several reasons that motivate our approach of using exclusively unlabeled data to esti- 
mate the risks. Labeled data may be unavailable due to privacy considerations where the predictors 
are constructed by organizations using training sets with private labels. For example, in medical 
diagnosis prediction, the predictors f),..., fg may be obtained by k different hospitals, each using a 
private internal labeled set. Following the training stage, each hospital releases its predictor to the 
public who then proceed to estimate R(f1),...,R(f;) using a separate unlabeled data set. 

Another motivation for using unlabeled data is domain adaptation where predictors that are 
trained on one domain, are used to predict data from a new domain from which we have only 
unlabeled data. For example, predictors are often trained on labeled examples drawn from the 
past but are used at test time to predict data drawn from a new distribution associated with the 
present. Here the labeled data used to train the predictors will not provide an accurate estimate due 
to differences in the test and train distributions. 

Another motivation is companies releasing predictors to clients as black boxes (without their 
training data) in order to protect their intellectual property. This is the situation in business analytics 
and consulting. In any case, it is remarkable that without labels we can still accurately estimate 
supervised risks. 

The collaborative nature of this diagnosis is especially useful for multiple predictors as the 
predictor ensemble {f1,..., fk} diagnoses itself. However, our framework is not restricted to a large 
k and works even for a single predictor with k = 1. It may further be extended to the case of active 
learning where classifiers are queried for specific data and the case of semi-supervised learning 
where a small amount of labeled data is augmented by massive unlabeled data. 

We proceed in the next section to describe the general framework and some important special 
cases. In Section 3 we discuss extensions to the general framework and in Section 4-5 we discuss the 
theory underlying our estimation process. In Section 6 we discuss practical optimization algorithms. 
Section 7 contains an experimental study. We conclude with a discussion in Section 8. 
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2. Unsupervised Risk Estimation Framework 


We adopt the framework presented in Section 1 with the added requirement that the predictors 
fi,---, fk are stochastic, that is, their prediction ĵ = f;(x) (conditioned on x) is a random variable. 
Such stochasticity occurs if the predictors are conditional models predicting values according to 
their estimated probability, that is, f; models a conditional distribution q; and predicts y’ with prob- 
ability g;(y’|x). 

As mentioned previously our goal is to estimate the risk associated with classification or re- 
gression models fi,..., fg based on unlabeled data x), ied x”) x p(x). The testing marginal and 
conditional distributions p(x), p(y|x) may differ from the distributions used at training time for the 
different predictors. In fact, each predictor may have been trained on a completely different training 
distribution, or may have been designed by hand with no training data whatsoever. We consider 
the predictors as black boxes and do not assume any knowledge of their modeling assumptions or 
training processes. 


At the center of our framework is the idea to define a parameter vector 8 € © which characterizes 
the risks R(f1),...,R(fx), that is, R(f;) = g;(8) for some function g; : © — R, j = 1,...,k. The 
parameter vector 8 is estimated from data by connecting it to the probabilities 


pi'y) = p(f; predicts y’| true label is y). 


More specifically, we use a plug-in estimate R(f;) = g;(6) where 6 maximizes the likelihood of 
(i 


the predictor outputs ¥ : = f;(x) with respect to the model pe($) = f pe(S|y) p(y) dy. The precise 


equations are: 





RFI), 9) =g (8 H,...,9)) where (1) 
pO SGP. 9f?) 
99 S f(x), 
Gm p) 9) = argmax £(0;9,...,9), (2) 
£(0;9,...,9) = ¥ log”, P) 3) 











The integral in (3) is over the unobserved label yl) associated with x“. It should be a continu- 
ous integral Loss for regression and a finite summation ym ()_, for classification. For notational 
simplicity we maintain the integral sign for both cases with the understanding that it is over a con- 
tinuous or discrete measure u, depending on the topology of Y. Note that (3) and its maximizer 
are computable without any labeled data. All that is required are the classifiers (as black boxes), 
unlabeled data x"!),...,x”), and the marginal label distribution p(y). 
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Besides being a diagnostic tool for the predictor accuracy, ĝmle 


gate fi,..., fj to predict the label of a new example x”®™ 


can be used to effectively aggre- 


anew 


ge = arg Max Pame (Y | fila), -a 


yey 
k 
= arg max p(y) | | pome(fj(2"") |y). (4) 
yey j=l 7 


As a result, our framework may be used to combine existing classifiers or regression models in a 
completely unsupervised manner. 

There are three important research questions concerning the above framework. First, what are 
the statistical properties of ĝmle and Å (consistency, asymptotic variance). Second, how can we 
efficiently solve the maximization problem (2). And third, how does the framework work in practice. 
We address these three questions in Sections 4-5, 6, 7 respectively, We devote the rest of the current 
section to examine some important special cases of (2)-(3) and consider some generalizations in the 
next section. 


2.1 Non-Collaborative Estimation of the Risks 


In the non-collaborative case we estimate the risk of each one of the predictors fi, ..., fg separately. 
This reduces the problem to that of estimating the risk of a single predictor, which is repeated k 
times for each one of the predictors. We thus assume in this subsection the framework (1)-(3) with 
k = 1 with no loss of generality. For simplicity we denote the single predictor by f rather than fi 


(i) 


and denote g = gı and $Ù = he . The corresponding simplified expressions are 


RFI 9.69) =o OPO esas), 


cael Gan eee a argmax ) log A pod ty) py) duly”) (5) 
i=1 


where $) = f(x). 
We consider below several important special cases. 


2.1.1 CLASSIFICATION 


Assuming l labels Y = {1,...,/}, the classifier f defines a multivariate Bernoulli distribution 
Poe(S|y) mapping the true label y to f 


Po(Sly) = 05y- (6) 


where 0 is the stochastic confusion matrix or noise model corresponding to the classifier f. In this 
case, the relationship between the risk R(f) and the parameter 0 is 


R(f) =1- } 6, pv). (7) 
yey 
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Equations (6)-(7) may be simplified by assuming a symmetric error distribution (Cover and Thomas, 


2005) 
i Ma 1—90 I(SAy) 
polly) = 0/0) G) , (8) 


R(f) =1-8 


where J is the indicator function and 6 € [0,1] is a scalar corresponding to the classifier accuracy. 
Estimating 0 by maximizing (5), with (6) or (8) substituting pe completes the risk estimation task. 
In the simple binary case / = 2, Y = {1,2} with the symmetric noise model (8) the loglikelihood 


gi) q — 00A.) p(y) 


£(8) = }_ log 
i=1 y 1 


iM» 


(0) 
may be shown to have the following closed form maximizer 


ĝme _ P9 =1) -m/n 


i 9 
2p(y=1)-1 K 


where m = |{i € {1,...,n} : § = 2}|. The estimator (9) works well in practice and is shown to be 


a consistent estimator in the next section (i.e., it converges to the true parameter value). In cases 
where the symmetric noise model (8) does not hold, using (9) to estimate the classification risk may 
be misleading. For example, in some cases (9) may be negative. In these cases, using the more 
general model (6) instead of (8) should provide more accurate results. We discuss this further from 
theoretical and experimental perspectives in Sections 4-5, and 7 respectively. 


2.1.2 REGRESSION 


Assuming a regression equation 
y=ax+€, e ~ N(0,1°) 


and an estimated regression model or predictor $ = a'x we have 


if 


j=ax=da'(y—e) = 8y — Oe 


where 0 = a'a~'. Thus, in the regression case the distribution pe(f|y) and the relationship between 
the risk and the parameter R(f) = g(@) are 


= $— 0 2 
poly) = (2701?) "/7exp ao (10) 
R(fly) = bias? (f) + Var (f) = (1-0)? +0, 

R(f) = 0°? + (1 -0E p90. 


Note that we consider regression as a stochastic estimator in that it predicts y = a'x + € or y|x ~ 
N(a'x,t’). 


1327 


DONMEZ, LEBANON AND BALASUBRAMANIAN 


Assuming p(y) =N (ty, 05) (as is often done in regression analysis) we have 





(i) _ (i = 292 ped) —1/2 (§—8y)? (v= by)? 
po) = | pol$»)p(r)dy = (282703) f oo( ee oe a 





1 (i) )2 o? 2 q2 și) 
= — Oxp u ) 2 A ea a a z 27l | + Q a E 2 
6, 2r(r +02) Te Gora o? | o2+r (2 +02) 
where we used the following lemma in the last equation. 


Lemma 1 (e.g., Papoulis, 1984) 


f Ae Be +CD dy = A is exp (C?/4B+ D) 


where A,B,C,D are constants that do not depend on x. 


In this case the loglikelihood simplifies to 


n (s(i))2 n (i) 2 
= 2 2 ae) ) 1 j My Lid 1 My 
(8) = —nlog (0,/2(22 +02) (z i) o2 Pie |O nF) 


which can be shown to have the following closed form maximizer 














n ali n »(i))\2 n (ai 
@mle — My LEI | WEI) | LL GOP 
n(t? +05) 


2n(t? +03) — 4n2(12 + 02)” 


where the two roots correspond to the two cases where 0 = a’ /a > 0 and 0 =a'/a < O. 

The univariate regression case described above may be extended to multiple explanatory vari- 
ables, that is, y = Ax + € where y,x,€ are vectors and A is a matrix. This is an interesting extension 
which falls beyond the scope of the current paper. 


2.1.3 NOISY GAUSSIAN CHANNEL 


In this case our predictor f corresponds to a noisy channel mapping a real valued signal y to its noisy 
version f. The aim is to estimate the mean squared error or noise level R(f) = E||y — ĵ||?. In this 
case the distribution pg(f|y) and the relationship between the risk and the parameter R(f) = g(8) 
are 





IERO) 
polly) = (20) "exp (ET) 
R(fly)=0°, 

RL) = PE p00. 


The loglikelihood and other details in this case are straightforward variations on the linear re- 
gression case described above. We therefore concentrate in this paper on the classification and linear 
regression cases. 
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As mentioned above, in both classification and regression, estimating the risks for k > 2 pre- 
dictors rather than a single one may proceed by repeating the optimization process described above 
for each predictor separately. That is R(f;) = g ;(6R*) where 6™°,...,6™° are estimated by max- 
imizing k different loglikelihood functions. In some cases the convergence rate to the true risks 
can be accelerated by jointly estimating the risks R(fi),...,R(fx) in a collaborative fashion. Such 
collaborative estimation is possible under some assumptions on the statistical dependency between 
the noise processes defining the k predictors. We describe below such an assumption followed by a 
description of more general cases. 


2.2 Collaborative Estimation of the Risks: Conditionally Independent Predictors 


We have previously seen how to estimate the risks of k predictors by separately applying (1) to each 
predictor. If the predictors are known to be conditionally independent given the true label, that is, 
Po(¥1,---,¥ely) = Il; pe, (¥;|y) the loglikelihood (3) simplifies to 


n k ; ; n f ; , 
(0) = $log [ [IpO po) du), where 9) = fie) (12) 
i=l j=l 


and pe, above is (6) or (8) for classification and (10) for regression. Maximizing the loglikelihood 
(12) jointly over 01, . . . , Ox results in estimators R(f),...,R(f;) that converge to the true value faster 
than the non-collaborative MLE (5) (more on this in Section 7). Equation (12) does not have a closed 
form maximizer requiring the use of iterative computational techniques. 


The conditional independence of the predictors is a much weaker condition than the indepen- 
dence of the predictors which is very unlikely to hold. In our case, each predictor f; has its own 
stochastic noise operator T;(r,s) = p(¥ = rly = s) (regression) or matrix [Tj],5 = pj(¥ = rly = s) 
(classification) where 7),...,7% may be arbitrarily specified. In particular, some predictors may be 
similar, for example, T; ~ T;, and some may be different, for example, T; Æ T;. The conditional 
independence assumption that we make in this subsection is that conditioned on the latent label y 
the predictions of the predictors proceed stochastically according to T,..., Tę in an independent 
manner. 


Figure 1 displays the loglikelihood functions (6) for three different data set sizes n = 100,250, 
500. As the size n of the unlabeled data grows the curves become steeper and ĝmle approach 0™°, 
Figure 2 displays a similar figure for k = 1 in the case of regression. 
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0.5 0.6 0.7 0.8 0.9 1 
Figure 1: A plot of the loglikelihood functions (0) in the case of classification for k = 1 (left, 
ome = 0.75) and k = 2 (right, 9" = (0.8,0.6) '). The loglikelihood was constructed 
based on random samples of unlabeled data with sizes n = 100,250,500 (left) and n = 250 
(right) and p(y = 1) = 0.75. In the left panel the y values of the curves were scaled so 
their maxima would be aligned. For k = 1 the estimators 6™'* (and their errors |6™!* — 
0.75|) for n = 100,250,500 are 0.6633 (0.0867), 0.8061 (0.0561), 0.765 (0.0153). As 
additional unlabeled examples are added the loglikelihood curves become steeper and 
their maximizers become more accurate and closer to 0™°®, 





loglikelinood 














Figure 2: A plot of the loglikelihood function (0) in the case of regression for k = 1 with 
em = 0.3, t= 1, uy = 0 and ©, = 0.2. As additional unlabeled examples are added the 


loglikelihood curve become steeper and their maximizers get closer to the true parameter 
9" resulting in a more accurate risk estimate. 
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In the case of regression (12) involves an integral over a product of k+ 1 Gaussians, assuming 
that y ~ N (uy, o2). In this case the integral in (12) simplifies to 








l l oo fk (500V Age dn)? , 
E =f ( ey =n) bg) 0 


jai OjtV 20 OyV 21 


i 7 i) C) 
1 oo 1 D k (i) ; 
= k+l I SXP — 2 Z - dy” 
(v2) ok 9; 2 Oy ja\ t 8; 
a(i) 2 a(i)\2 
2. i f y j i y Gj) 
JZ- exp (-4 (4 T 4) (yl N H (4 H Ei 7 3 (4 +E% Hy D 
= k+l 
(Vm) oy TK, 


2 
-1/2 sP l 
vajata t)] (Sta) aop a 
exp 
(Vm 10, TT 8; 2(4+4) 262 262 


2 2 
o5 T 























(13) 


where the last equation was obtained using Lemma 1 concerning Gaussian integrals. Note that 
this equation does not have a closed form maximizer requiring the use of iterative computational 
techniques. 


2.3 Collaborative Estimation of the Risks: Conditionally Correlated Predictors 


In some cases the conditional independence assumption made in the previous subsection does not 
hold and the factorization (12) is violated. In this section, we discuss how to relax this assumption 
in the classification case. A similar approach may also be used for regression. We omit the details 
here due to notational clarity. 

There are several ways to relax the conditional independence assumption. Most popular, per- 
haps, is the mechanism of hierarchical loglinear models for categorical data (Bishop et al., 1975). 
For example, generalizing our conditional independence assumption to second-order interaction 
log-linear models we have 


log p(¥1,-.-Ikly) = 05+ Y Bis + Dios (14) 
i<j 


where the following ANOVA-type parameter constraints are needed (Bishop et al., 1975) 
0= » Bis.y Vir, (15) 


0= Esso = Emus Vi, j,y. 
fj 


The B parameters in (14) correspond to the order-1 interaction between the variables ĵ1,... , fk, 
conditioned on y. They correspond to the 9; in the independent formulation (6)-(8). The y parameters 
capture two-way interactions which do not appear in the conditionally independent case. Indeed, 
setting Y; js; 5;y = 0 retrieves the independent models (6)-(8). 
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In the case of classification, the number of degrees of freedom or free unconstrained parameters 
in (14) depends on whether the number of classes is 2 or more and what additional assumptions exist 
on B and y. For example, assuming that the probability of fj, f; making an error depends on the true 
class y but not on the predicted classes ¥;,9; results in a k + k? parameters. Relaxing that assumption 
but assuming binary classification results in 2k + 4k” parameters. The estimation and aggregation 
techniques described in Section 2.1 work as before with a slight modification of replacing (6)-(8) 
with variations based on (14) and enforcing the constraints (15). 

Equation (14) captures two-way interactions but cannot model higher order interactions. How- 
ever, three-way and higher order interaction models are straightforward generalizations of (14) cul- 
minating in the full loglinear model which does not make any assumption on the statistical depen- 
dency of the noise operators 7),...,7%. However, as we weaken the assumptions underlying the 
loglinear models and add higher order interactions the number of parameters increases adding to 
the difficulty in estimating the risks R(fi),...,R(fx)- 

In our experiments on real world data (see Section 7), it is often the case that maximizing the 
loglikelihood under the conditionally independent assumption (12) provides adequate accuracy and 
there is no need for the more general (14)-(15). Nevertheless, we include here the case of loglinear 
models as it may be necessary in some situations. 


3. Extensions: Missing Values, Active Learning, and Semi-Supervised Learning 


In this section, we discuss extensions to the current framework. Specifically, we consider extending 
the framework to the cases of missing values, active and semi-supervised learning. 

Occasionally, some predictors are unable to provide their output over specific data points. That 
is assuming a data set x)... ,x each predictor may provide output on an arbitrary subset of the 
data points { f;(x) : i € Sj}, where S; C {1,...,n}, j=1,...,k. 

Commonly referred to as a missing value situation, this scenario may apply in cases where dif- 
ferent parts of the unlabeled data are available to the different predictors at test time due to privacy, 
computational complexity, or communication cost. Another example where this scenario applies is 
active learning where operating f; involves a certain cost c j > 0 and it is not advantageous to operate 
all predictors with the same frequency for the purpose of estimating the risks R(f),...,R(f,). Such 
is the case when f; corresponds to judgments obtained from human experts or expensive machinery 
that is busy serving multiple clients. Active learning fits into this situation with S; denoting the set 
of selected data points for each predictor. 

We proceed in this case by defining indicators Bj; denoting whether predictor j is available 
to emit fj (x), The risk estimation proceeds as before with the observed likelihood modified to 
account for the missing values. 

In the case of collaborative estimation with conditional independence, the estimator and log- 
likelihood become 


6me — arg max (8), 
to) 
=F] S auc 16 
(@)=} log } POL ok du) (16) 
i=l Bae 


-Fio Z ff pols}?.....5. pO du any) 


i=l r: Bri=0 
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where pe may be further simplified using the non-collaborative approach, or using the collaborative 
approach with conditional independence or loglinear model assumptions. 

In the case of semi-supervised learning a small set of labeled data is augmented by a large set 
of unlabeled data. In this case our framework remains as before with the likelihood summing over 
the observed labeled and unlabeled data. For example, in the case of collaborative estimation with 
conditional independence we have 


n k m k 
(8) = roe | TJoa 6P yo duty) È los] [ ro, (5) >) 0). 
i=l Y j=l i=n+1 j=l 
The different variations concerning missing values, active learning, semi-supervised learning, 
and non-collaborative or collaborative estimation with conditionally independent or correlated noise 
processes can all be combined in different ways to provide the appropriate likelihood function. This 
provides substantial modeling flexibility. 


4. Consistency of 6™® and R(f;) 


In this and the next section we consider the statistical behavior of the estimator ĝmle defined in 
(2) and the risk estimator R(f;) = g;(ĝ™°) defined in (1). The analysis is conducted under the 
assumption that the vectors of observed predictors outputs $ = (90? ates 90) are iid samples from 
the distribution 


po(S) = polS1,---.5e) = L poi- skly) pO) duly). 


We start by investigating whether estimator ĝmle in (2) converges to the true parameter value. 
More formally, strong consistency of the estimator ĝmle = ô, ah s), pO). pO) x Pe is 
defined as strong convergence of the estimator to Oo as n — œ (Ferguson, 1996) 

lim 6™°(p(),..., 9”) = Oo with probability 1. 

n— oo 
In other words as the number of samples n grows, the estimator will surely converge to the true 
parameter 69 governing the data generation process. 

Assuming that the risks R(fj) = g;(8) are defined using continuous functions g j, strong consis- 
tency of 6" implies strong convergence of R(f;) to R(f;). This is due to the fact that continuity 
preserves limits. Indeed, as the g; functions are continuous in both the classification and regression 
cases, strong consistency of the risk estimators R(f j) reduces to strong consistency of the estimators 
ĝmle, 

It is well known that the maximum likelihood estimator is often strongly consistent. Consider, 
for example, the following theorem. 


Proposition 2 (e.g., Ferguson, 1996) Let $®,...,5™ © po, B0 € ©. If the following conditions 


hold 
1. © is compact (compactness) 
2. paĝ) is upper semi-continuous in 8 for all ¥ (continuity) 
3. There exists a function K($) such that Ep, |K ($)| < œ% (boundedness) 


and log po(¥) — log pa ($) < K(¥) VS VƏ 

4. For all ® and sufficiently small p > 0, supjg_¢)<p Pø () is (measurability) 
measurable in ¥ 

5. Po = Po, ™ 9 = 9% (identifiability) 


1333 


DONMEZ, LEBANON AND BALASUBRAMANIAN 


Qe 


then the maximum likelihood estimator is strongly consistent, that is, — 09 as n — œ with 


probability 1. 


Note that pe(¥) in the proposition above corresponds to fy pe(S|y) p(y) du(y) in our framework. 
That is the MLE operates on the observed data or predictor output $(!),..., 9 that is sampled iid 
from the distribution pe, ($) = fy Pos (Sly) p(y) du). 

Of the five conditions above, the last condition of identifiability is the only one that is truly prob- 
lematic. The first condition of compactness is trivially satisfied in the case of classification. In the 
case of regression it is satisfied assuming that the regression parameter and model parameter are fi- 
nite and a Æ 0 as the estimator 6™© will eventually lie in a compact set. The second condition of con- 
tinuity is trivially satisfied in both classification and regression as the function fy pe(¥|y) p(y) du) 
is continuous in 8 once ĵ is fixed. The third condition is trivially satisfied for classification (finite 
valued y). In the case of regression due to conditions 1,2 (compactness and semi-continuity) we 
can replace the quantifier YO with a particular value 6’ € © representing worst case situation in the 
bound of the logarithm difference. Then, the bound K may be realized by the difference of log 
terms (with respect to that worst case 8’) whose expectation converges to the KL divergence which 
in turn is never co for Gaussian distributions or its derivatives. The fourth condition of measurability 
follows as pe is specified in terms of compositions, summations, multiplications, and point-wise 
limits of well-known measurable functions. 

The fifth condition of identifiability states that if pọ (f) and pe, (f) are identical as functions, that 
is, they are identical for every value of f, then necessarily © = 09. This condition does not hold in 
general and needs to be verified in each one of the special cases. 

We start with establishing consistency in the case of classification where we rely on a symmetric 
noise model (8). The non-symmetric case (6) is more complicated and is treated afterwards. We 
conclude the consistency discussion with an examination of the regression case. 


4.1 Consistency of Classification Risk Estimation 


Proposition 3 Let f),..., fk be classifiers f; : X — Y, || =L, with conditionally independent noise 
processes described by (8). If the classifiers are weak learners, that is, 1/1 < 1—err(fi) < 1 and 
p(y) is not uniform the unsupervised collaborative diagnosis model is identifiable. 


Corollary 4 Let f\,..., f; be classifiers f;: X —> Y with |Y| = l and noise processes described by 
(8). If the classifiers are weak learners, that is, 1/1 < 1—err(f;) < 1, and p(y) is not uniform the 
unsupervised non-collaborative diagnosis model is identifiable. 


Proof Proving identifiability in the non-collaborative case proceeds by invoking Proposition 3 
(whose proof is given below) with k = 1 separately for each classifier. The conditional indepen- 
dence assumption in Proposition 3 becomes redundant in this case of a single classifier, resulting in 
identifiability of pe,(¥;) foreach j = 1,...,k E 


Corollary 5 Under the assumptions of Proposition 3 or Corollary 4 the unsupervised maximum 
likelihood estimator is consistent, that is, 


P (lim Gre (p(t), oe sy) = (07%, di aaee) =1. 


n— oo 
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Consequentially, assuming that R( fj) = g;(0), j = 1,...,k with continuous gj; we also have 


P (tim R(F;39,....9) =R(f;), Yj= kaa) =i 


n—-oo 


Proof Proposition 3 or Corollary 4 establishes identifiability, which in conjunction with Proposi- 
tion 2 proves the corollary. a 


Proof (for Proposition 3) We prove identifiability by induction on k. In the base case of k = 1, we 
have a set of / equations, corresponding toi=1,2...1, 





v 
D 
— 
= 
5 
II 
a 
x 
II 


po=i) + [Eo - i) ae 
ivi 








p(y =i) + (1— p(y =i)) 


@ (p(y =i) —1)+1—p(y=i) 
(1—1) 





from which we can see that if 7 Æ 0 and p(y =i) Æ 1/1 then pọ(f1) Æ pn(ĵı). This proves identifi- 
ability for the base case of k = 1. 

Next, we assume identifiability holds for k and prove that it holds for k+ 1. We do so by 
deriving a contradiction from the assumption that identifiability holds for k but not for k+ 1. We 
denote the parameters corresponding to the k labelers by the vectors 0,7 € [0, 1] and the parameters 
corresponding the additional k + 1 labeler by 944.1, 441. 


In the case of k classifiers we have 


G(4j,9®) 


iM 


l 
Pali,- = ¥ poi, -y = Dp =) = 
i=1 


where 





G(4;,8) = p(y =) [o II a 


ja jga l7) 
AZ {je {1,2... k}: =i}. 


Note that the 4),...,4) form a partition of {1,...,k}, that is, they are disjoint and their union is 
{1,..., k}. 

In order to have unidentifiability for the k + 1 classifiers we need (8,041) Æ (N, Nz+1) and the 
following / equations (corresponding to ¥,,; = 1,2,...,/) to hold for any $1,...,%, which corre- 
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sponds to any partition £4, ..., A 




















1 — Oxy 1-—nNe 
04116 ( 41,0) + E7 Y (4,8) = ne GAisn) + CMY © Gan), 
(/—1) iAl (7-1) iAl 
1 — k+ 1— Nk 
Oz+1G(%,0) nl. (1 Bes) Ł G(A;,0) = Ne G(%,N) I: =ne) Ł G(Ai,n), 
(l-1) iD (l-1) iD 
1-0 
04 41G(Ay,0) + LPH F GA, 0) = mee Gln) + Sey ot (An). 
(l-1) mz (l-1) mz) 
We consider two cases in which (0, 0+1) Æ (M, Nz+1): (a) 0 Æ n, and (b) 0 = N, Ok+1 Æ Nk+1- 
In the case of (a) we add the / equations above which marginalizes +1 out of po(S1,..-, k, Sk+1) 


and pn (i, ..-, fk, 341) to provide 
l 


l 
$ G( 4,8) = Lalan) 


i=l i=l 
which together with 0 Æ n contradicts the identifiability for the case of k classifiers. 
In case (b) we have from the / equations above 





9x41G(Ar, 8) + — 7 [Zero A0) — «ta. 





I 
=n G(A4,N) + e (ze (Ain) — cian) 


for any t € {1,...,/} which simplifies to 


0= eni -nen (164.8) 8) -FG Ai, ») t=1,...,k. 


i=1 


As we assume at this point that 0,4; Æ Ng+1 the above equality entails 


l 
=) G(A;,8). (17) 
i=l 
We show that (17) cannot hold by examining separately the cases p(y =t) > 1/1 and p(y =t) < 1/1. 
Recall that there exists a t for which p(y = t) # 1/1 since the proposition requires that p(y) is not 


uniform. 
If p(y =t) > 1/1 we choose 4, = {1,...,k} and obtain 











k ; k 
py=) [19 = Leo =dTI5 “+ py=n) 16) 
j=l ift j=l j=l 
k =p 
(po =)= 0 -205D T 
j= j= 
k _ = k = 
po =V]; =" ae “Tl i 
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which cannot hold as the term on the left hand side is necessarily larger than the term on the right 
hand side (if p(y = t) > 1// and 0; > 1/1). In the case p(y =t) < 1/1 we choose A, = {1,...,k}, 
s Æ t to obtain 


= 
l 

fap) 
ce 
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= 
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II 
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DP 


S. 

Il 
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`. 

Il 
en 


which cannot hold as the term on the left hand side is necessarily smaller than the term on the right 
hand side (if p(y =t) < 1/l and 0; > 1/1). 

Since we derived a contradiction to the fact that we have k-identifiability but not k + 1 identifia- 
bility, the induction step is proven which establishes identifiability for any k > 1. a 


The conditions asserted above that p(y) 4 1/1 and 1/1 < 1 —err(f;) < 1 are intuitive. If they 
are violated a certain symmetry may emerge which renders the model non-identifiable and the MLE 
estimator not consistent. 

In the case of the non-collaborative estimation for binary classification with the non-symmetric 
noise model, the matrix © in (6) is a 2 x 2 matrix with two degrees of freedom as each row sums to 
one. In particular we have 011 = pe(f = lly = 1), 812 = po(¥ = 1|y = 2), O21 = pof = 2|y = 1), 
O02 = po (f = 2|y = 2) with the overall risk R( f) = 1 — 011 p(y = 1) —®22p(y = 2). Unfortunately, the 
matrix © is not identifiable in this case and neither is the scalar parameter 011 p(y = 1) + ®22p(y = 2) 
that can be used to characterize the risk. 

We can, however, obtain a consistent estimator for O (and therefore for R(f)) by first showing 
that the parameter 011 p(y = 1) — 922p(y = 2) is identifiable and then taking the intersection of two 
such estimators. 


Lemma 6 In the case of the non-collaborative estimation for binary classification with the non- 
symmetric noise model and p(y) # 0, the parameter 81 p(y = 1) — 922p(y = 2) is identifiable. 


Proof For two different parameterizations 8,7 we have 


pol = 1) = p(y = 1/011 +0 — p(y = 1) (1 — 022), (18) 

po = 2) = p(y = 1)(1— 811) + (1 — p(y = 1)) 822 (19) 
and 

Ph = 1) = 


ply = 1)nu +0 — p(y =1)) —n22), (20) 
Pa(¥ = 2) = p(y = 1) —nn) + 1 — p(y = 1)) nz. (21) 


Equating the two Equations (18) and (20) we have 
pO = 1) (811 + O22) +1 — p(y = 1) — Ox = p(y = 1) (M1 +122) +1- p0 =1)-n2 


pO = 1)011 — (1— p(y = 1))022 = py = Inu — (1 — p(y = 1))n2 
PO = 1)811 — p(y = 2)822 = p(y = 1)ni — ply = 2)n22 
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Similarly, equating Equation (19) and Equation (21) also results in p(y = 1)01; — p(y = 2)022 = 
pO = ln — p(y =2)n2. As a result, we have 








Pe=Py > P(y=1)011— p(y =2)822 = p(y = 1M1 — p(y = 2)n2. 


The above lemma indicates that we can use the maximum likelihood method to obtain a consis- 
tent estimator for the parameter 011 p(y = 1) — 922 p(y = 2). Unfortunately the parameter 011 p(y = 
1) — 822p(y = 2) does not have a clear probabilistic interpretation and does not directly characterize 
the risk. As the following proposition shows we can obtain a consistent estimator for the risk R(f) 
if we have two populations of unlabeled data drawn from distributions with two distinct marginals 


pı(y) and po(y). 


Proposition 7 Consider the case of the non-collaborative estimation of binary classification risk 
with the non-symmetric noise model. If we have access to two unlabeled data sets drawn indepen- 
dently from two distributions with different marginals, that is, 


xD... © pi(x) = ¥ ply) p10), 
y 


POL & po(x) =¥ p(aly) poly) 
y 


we can obtain a consistent estimator for the classification risk R(f). 


Proof Operating the classifier f on both sets of unlabeled data we get two sets of observed clas- 
sifier outputs §),...,9, PO ,..., 9 where $Ù © X, po(Sly)pi(y) and FO ~ E, po(S|y)p2(). 
In particular, note that the marginal distributions p;(y) and p(y) are different but the parameter 
matrix 0 is the same in both cases as we operate the same classifier on samples from the same class 
conditional distribution p(x|y). 

Based on Lemma 6 we construct a consistent estimator for pi(y = 1)011 — pi(y = 2)022 by 
maximizing the likelihood of 9). 9), Similarly, we construct a consistent estimator for p2(y = 
1)01; — p2(y = 2)®22 by maximizing the likelihood of §“),...,9. Note that pi(y = 1)01; — 
Pi(y = 2)022 and po(y = 1)011 — p2(y = 2)822 describe two lines in the 2-D space (011,022). Since 
the true value of 011,022 represent a point in that 2-D space belonging to both lines, it is neces- 
sarily the intersection of both lines (the lines cannot be parallel since their linear coefficients are 
distributions which are assumed to be different). 

As n and m increase to infinity, the two estimators converge to the true parameter values. As a 
result, the intersection of the two lines described by the two estimators converges to the true values 
of (611,922) thus allowing reconstruction of the matrix 6 and the risk R(f). a 


Clearly, the conditions for consistency in the asymmetric case are more restricted than in the 
symmetric case. However, situations such as in Proposition 7 are not necessarily unrealistic. In 
many cases it is possible to identify two unlabeled sets with different distributions. For example, if 
y denotes a medical condition, it may be possible to obtain two unlabeled sets from two different 
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hospitals or two different regions with different marginal distribution corresponding to the frequency 
of the medical condition. 

As indicated in the previous section, the risk estimation framework may be extended beyond 
non-collaborative estimation and collaborative conditionally independent estimation. In these ex- 
tensions, the conditions for identifiability need to be determined separately, in a similar way to 
Corollary 4. A systematic way to do so may be obtained by noting that the identifiability equations 


O= po(S1,--- 3k) — P1- --9k) VIa,- +. De 


is a system of polynomial equations in (6,n). As a result, demonstrating lack of identifiability 
becomes equivalent to obtaining a solution to a system of polynomial equations. Using Hilbert’s 
Nullstellensatz theorem we have that a solution to a polynomial system exists if the polynomial 
system defines a proper ideal of the ring of polynomials (Cox et al., 2006). As k increases the 
chance of identifiability failing decays dramatically as we have a system of J‘ polynomials with 2k 
variables. Such an over-determined system with substantially more equations than variables is very 
unlikely to have a solution. 

These observations serve as both an interesting theoretical connection to algebraic geometry as 
well as a practical tool due to the substantial research in computational algebraic geometry. See 
Sturmfels (2002) for a survey of computational algorithms and software associated with systems of 
polynomial equations. 


4.2 Consistency of Regression Risk Estimation 
In this section, we prove the consistency of the maximum likelihood estimator 6™€ in the regression 


case. As in the classification case our proof centers on establishing identifiability. 


Proposition 8 Let f\,..., fg be regression models f;(x) = aix with y ~ N (uy, 95), y=axt+e. Assum- 
ing that a 4 0 the unsupervised collaborative estimation model assuming conditionally independent 
noise processes (12) is identifiable. 


Corollary 9 Let fi,..., fk be regression models f;(x) = aix with y ~ N (uy, 9), y=ax+e. Assuming 
that a £ 0 the unsupervised non-collaborative estimation model (12) is identifiable. 


Proof Proving identifiability in the non-collaborative case proceeds by invoking Proposition 8 
(whose proof is given below) with k = 1 separately for each regression model. The conditional 
independence assumption in Proposition 8 becomes redundant in this case of a single predictor, re- 
sulting in identifiability of pe,($;) for each j = 1,...,k. E 


Corollary 10 Under the assumptions of Proposition 8 or Corollary 9 the unsupervised maximum 
likelihood estimator is consistent, that is, 


P (tim êg GO, y) = (00,0) = 1. 


n— o 


Consequentially, assuming that R( fj) = g;(0), j = 1,...,k with continuous g; we also have 


P| imna Oe R W enk) =t 
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Proof Proposition 8 or Corollary 9 establish identifiability, which in conjunction with Proposition 2 
completes the proof. a 


Proof (of Proposition 8). 
We will proceed, as in the case of classification, with induction on the number of predictors k. 
In the base case of k = 1 we have derived pg, (1) in Equation (11). Substituting in it fı = 0 we get 











1 Ms T 
Pa 1 =0) = ep| -5| z1] 
l 01 ,/2n(t? +02) 205 (oy +T 
1 e T? 
Py, (91 =0)= oo( (ata!) 
l N14 /20(12 +02) 205 \oy +t? 


The above expression leads to 01 # N1 = Po, (1 = 0) # Py, (¥1 = 0) which implies identifiability. 

In the induction step we assume identifiability holds for k and we prove that it holds also for k+ 1 
by deriving a contradiction to the assumption that it does not hold. We assume that identifiability 
fails in the case of k + 1 due to differing parameter values, that is, 


P (00x41) (P15 +++ Sk Se) = Pama) V1s--- Te Tet) VIZ ER F=1,...,44+1 (22) 


with (0,041) Æ (N, Ng+1) where 6,n € R*. There are two cases which we consider separately: (a) 
6 £n and (b) 9= 7. 

In case (a) we marginalize both sides of (22) with respect to $,4; which leads to a contradiction 
to our assumption that identifiability holds for k 


oo 


‘| P0,011) (V1 +++ Dkr k1 )d kt al Paman iIk De AIH 
Polis- -3 fk) = Py(V1,--+ Ik): 
In case (b) 8 = n and 0,4; Æ Nk+1. Substituting fı = --- = k41 =O in (22) (see (13) for a 
derivation) we have 


Poo) (91 =0,..., k41 = 0) = Poma) 1 =0,..., k41 = 0) 














or 
-1/2 2 
1( 1. k+l Ly 
vala (a+) x (a) g 
Jan k 2 
TT (V20) I-10; 2 (4 + et) 205 
-1/2 Re 
alge) C 
= ; exp 
THT (V2) Money M-N 2 (è + t) 265 
which cannot hold if 0 = n but 044.1 A Nk+1- a 
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5. Asymptotic Variance of ĝmle and R 


A standard result from statistics is that the MLE has an asymptotically normal distribution with 
mean vector 6" and variance matrix (nJ(0""°))-!, where J(®) is the r x r Fisher information 
matrix 


J(0) = E {V log pe(5)(V log pe()) '} 


with V log pe(S) represents the r x 1 gradient vector of log pe(¥) with respect to ©. Stated more 
formally, we have the following convergence in distribution as n — œ (Ferguson, 1996) 


Vn (ôme — 99) ~> N(0, J7! (0™°)). (23) 


It is instructive to consider the dependency of the Fisher information matrix, which corresponds 
to the asymptotic estimation accuracy, on n, k, p(y), 0™°. 
In the case of classification considering (8) with k = 1 and Y = {1,2} it can be shown that 


a(2a— 1)? (2a—1)?(a—1) 
(0(2a—1)—a+1)? (a—@(2a—1))? 


where & = P(y = 1). As Figure 3 (right) demonstrates, the asymptotic accuracy of the MLE (as 
indicated by J) tends to increase with the degree of non-uniformity of p(y). Recall that since identi- 
fiability fails for a uniform p(y) the risk estimate under a uniform p(y) is not consistent. The above 
derivation (24) is a quantification of that fact reflecting the added difficulty in estimating the risk as 
we move closer to a uniform label distribution œ — 1/2. The dependency of the asymptotic accu- 
racy on 6""* is more complex, tending to favor 6" values close to 1 or 0.5. Figure 3 (left) displays 
the empirical accuracy of the estimator as a function of p(y) and 6™° and shows remarkable simi- 
larity to the contours of the Fisher information (see Section 7 for more details on the experiments). 
In particular, whenever the estimation error is high the asymptotic variance of the estimator is high 
(or equivalently, the Fisher information is low). For instance, the top contours in the left panel have 
smaller estimation error on the top right than in the top left. Similarly, the top contours in the right 
panel have smaller asymptotic variance on the top right than on the top left. We thus conclude that 
the Fisher information provides practical, as well as theoretical insight into the estimation accuracy. 

Similar calculations of J(6""°) for collaborative classification case or for the regression case 
result in more complicated but straightforward derivations. It is important to realize that consistency 
is ensured for any identifiable 6°, p(y). The value (J(0"°))~! is the constant dominating that 
consistency convergence. 

A similar distributional analysis can be derived for the risk estimator. Applying Cramer’s theo- 
rem (Ferguson, 1996) to R(f;) = g)(8™*), j=1,...,k and (23) we have 


ValR(f) — R(f)) ~ N (0, V9(9™)J(0™) V9)" ) 





J(8) = (24) 


where R(f),R(f) are the vectors of true risk and risk estimates for the different predictors f,,..., fk 
and Vg(0™°) is the Jacobian matrix of the mapping g = (g1,...,g%) evaluated at 6'™°. 

For example, in the case of classification with k = 1 we have R(f;) = 1 — 9; and the Jacobian 
matrix is —1, leading to an identical asymptotic distribution to that of the MLE (23)-(24) 


-1 
vieu- — (a, ( leu Hert) ) 





(0(2a—1)—-a+1)? (a—0(2a—1))? 
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6. Optimization Algorithms 


Recall that we obtained closed forms for the likelihood maximizers in the cases of non-collaborative 
estimation for binary classifiers and non-collaborative estimation for one dimensional regression 
models. The lack of closed form maximizers in the other cases necessitates iterative optimization 
techniques. 

One class of technique for optimizing nonlinear loglikelihoods is the class of gradient based 
methods such as gradient descent, conjugate gradients, and quasi Newton methods. These tech- 
niques proceed iteratively following a search direction; they often have good performance and are 
easy to derive. The main difficulty with their implementation is the derivation of the loglikelihood 
and its derivatives. For example, in the case of collaborative estimation of classification (/ > 2) with 
symmetric noise model and missing values the loglikelihood gradient is 


ZPO ORE X Tpz; hol © = yO) -e (0- 190) 1 — 8) 107 = 


yO r: Bri=0 s (i) 








aÈ 


D) (i r5 Ay 
j= g0? =) 1-0, (Sp Ay) 
pi = Yp i 


Similar derivations may be obtained in the other cases in a straightforward manner. 

An alternative iterative optimization technique for finding the MLE is expectation maximization 
(EM). The derivation of the EM update equations is again relatively straightforward. For example 
in the above case of collaborative estimation of classification (l > 2) with symmetric noise model 
and missing values the EM update equations are 


Eyo PO) Engu Leto Mp-1 Api 


k . F 
atte arena Y ES.) Fo tow 65/0") 
© r:Bi=0 gl j=l 
1# i i a(i i 
=- EE L LOPO? =»), 
i=1 yl) r:B,i=0 99 





p(y) TE, pj(9? |, 00) 
Lo Erg,- 2 PO) T= i pj6P y0, 00) 





g 0P ) = 


where q is the conditional distribution defining the EM bound over the loglikelihood function. 

If all the classifiers are always observed, that is, B,; = 1 Vr,i Equation (16) reverts to (12), and 
the loglikelihood and its gradient may be efficiently computed in O(nlk*). In the case of missing 
classifier outputs a naive computation of the gradient or EM step is exponential in the number of 
missing values R = max; }, Bn. This, however, can be improved by careful dynamic programming. 
For example, the nested summations over the unobserved values in the gradient may be computed 
using a variation of the elimination algorithm in O(nlk?R) time. 


7. Empirical Evaluation 


We start with some experiments demonstrating our framework using synthetic data. These experi- 
ments are meant to examine the behavior of the estimators in a controlled setting. We then describe 
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Figure 3: Left: Average value of |ĝ™!e — 6'*| as a function of 0"™™° and p(y = 1) for k = | classifier 
and n = 500 (computed over a uniform spaced grid of 15 x 15 points). The plot illustrates 
the increased accuracy obtained by a less uniform P(y). Right: Fisher information J (0) 
for k = 1 as a function of 6° and P(y). The asymptotic variance of the estimator is 
J~'(®) which closely matches the experimental result in the left panel. 


some experiments using several real world data sets. In these experiments we examine the behav- 
ior of the estimators in an uncontrolled setting where some of the underlying assumptions may be 
violated. In most of the experiments we consider the mean absolute error (mae) or the 4; error as a 
metric that measures the estimation quality 


1 


k 
mae(6™°, girue) = £ joe = gms f 
i=1 


a| 


In the non-collaborative case (which is equivalent to the collaborative case with k = 1) this translates 
into the absolute deviation of the estimated parameter from the true parameter. 

In Figure 3 (left) we display mae(ĝ™!e, e) for classification with k = 1 as a function of 0™° 
and p(y) for n = 500 simulated data points. The estimation error, while overall relatively small, 
decays as p(y) diverges from the uniform distribution. The dependency on 6""° indicates that the 
error is worst for 6""° around 0.75 and it decays as |6""° — 0.75| increases with a larger decay 
attributed to higher 6'"°. These observations are remarkably consistent with the developed theory 
as Figure 3 (right) shows by demonstrating the value of the inverse asymptotic variance J(®) which 
agrees nicely with the empirical measurement in the left panel. 

Figure 4 (left) contains a scatter plot contrasting values of @""° and ĝ™! for k = 1 classifier and 
p(y = 1) = 0.8. The estimator was constructed based on 500 simulated data points. We observe 
a symmetric Gaussian-like distribution of estimated values 6™*, conditioned on specific values of 
o"e, This is in agreement with the theory predicting an asymptotic Gaussian distribution for the 
mle, centered around the true value 6""°. A similar observation is made in Figure 5 (left) which 
contains a similar scatter plot in the regression case (k = 1, 6, = 1, n = 1000). In both figures, the 
striped effect is due to selection of 6""° over a discrete grid with a small perturbation for increased 
visibility. Similar plots of larger and smaller n values (not shown) verify that the variation of ĝmle 
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Figure 4: Left: Scatter plot contrasting the true and predicted values of 8 in the case of a single clas- 
sifier k = 1, p(y = 1) = 0.8, and n = 500 unlabeled examples. The displayed points were 
perturbed for improved visualization and the striped effect is due to empirical evaluation 
over a discrete grid of 6° values. Right: mae(6™*,6"™*) as a function of the number 
of unlabeled examples for different number of classifiers (6/"° = p(y = 1) = 0.75) in the 
collaborative case. The estimation error decreases as more classifiers are used due to the 
collaborative nature of the estimation process. 


around 6""° decreases as n increases. This agrees with the theory that indicates a O(n!) rate of 
decay for the variance of the asymptotic distribution. 

Figures 4 and 5 (right) show the mae(6™*, 6°) for various k values in classification and re- 
gression, respectively. In classification, 6™e was obtained by sampling data from p(y = 1) = 
0.75 = 61° Vi. In regression, the data was sampled from the regression equation with 6"° = 1 
and p(y) = N(0,1). In both cases, the mae error decays with n as expected from the consistency 
proof and with k as a result of the collaborative estimation effect. 

To further illustrate the effect of the collaboration on the estimation accuracy, we estimated the 
error rates individually (non-collaboratively) for 10 predictors and compared their mae to that of 
the collaborative estimation case in Figure 6. This shows that each of the classifiers have a similar 
mae curve when non-collaborative estimation is used. However, all of these curves are higher than 
the collaborative mae curve (solid black line in Figure 6) demonstrating the improvement of the 
collaborative process. 

We compare in Figure 7 the proposed unsupervised estimation framework with supervised es- 
timation that takes advantage of labeled information to determine the classifier accuracy. We con- 
ducted this study using equal number of examples for both supervised and unsupervised cases. 
Clearly, this is an unfair comparison if we assume that labeled data is unavailable or is difficult to 
obtain. The unsupervised estimation does not perform as well as the supervised version especially 
in general. Nevertheless, the unsupervised estimation accuracy improves significantly with increas- 
ing number of classifiers and finally reaches the performance level of the supervised case due to 
collaborative estimation. 

In Figure 8 we report the effect of misspecification of the marginal p(y) on the estimation 
accuracy. More specifically, we generated synthetic data using a true marginal distribution but 
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Figure 5: Left: Scatter plot contrasting the true and predicted values of O in the case of a single 
regression model k = 1, ©, = 1, and n = 1000 unlabeled examples. The displayed points 
were perturbed for improved visualization and the striped effect is due to empirical eval- 
uation over a discrete grid of 0° values. Right: mae(6™*,@'™°) as a function of the 
number of unlabeled examples for different number of regression models (6/"* = ©, = 1) 
in the collaborative case. The estimation error decreases as more regression models are 
used due to the collaborative nature of the estimation process. 
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Figure 6: Comparison of collaborative and non-collaborative estimation for k = 10 classifiers. 
mae(6™*, ote) as a function of n is reported for 6!" = 0.75 Vk; and P(y = 1) = 0.75. 
The colored lines represent the estimation error for each individual classifier and the solid 
black line represents the collaborative estimation for all classifiers. The estimation con- 
verges to the truth faster in the collaborative case than in the non-collaborative case. 
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Figure 7: Comparison of supervised and unsupervised estimation for different values of classifiers 
with k = 1,3,5,10. Supervised estimation uses the true labels to determine the accuracy 
of the classifiers whereas in the unsupervised case the estimation proceeds according to 
the collaborative estimation framework. Despite the fact that the supervised case uses 
labels the unsupervised framework reaches similar levels by increasing the number of 
classifiers. 


estimated the classifier accuracy on this data assuming a misspecified marginal. Generally, the 
estimation framework is robust to small perturbations while over-specifying tends to hurt less than 
under-specifying (misspecification closer to uniform distribution). 

Figure 9 shows the mean prediction accuracy for the unsupervised predictor combination scheme 
in (4) for synthetic data. The left panel displays classification accuracy and the right panel displays 
the regression accuracy as measured by 1 — 1 m (new — yrew)?, The graphs show that in both 
cases the accuracy increases with k and n in accordance with the theory and the risk estimation 
experiments. The parameter 6/""° was chosen uniformly in the range (0.5,1), and P(y = 1) = 0.75 
for classification and 6®™™° = 0.3, p(y) = N(0, 1) in the case of regression. 

We also experimented with the natural language understanding data set introduced in Snow 
et al. (2008). This data was created using the Amazon Mechanical Turk (AMT) for data annotation. 
AMT is an online tool that uses paid employees to complete small labeling and annotation tasks. 
We selected two binary tasks from this data: the textual entailment recognition (RTE) and temporal 
event recognition (TEMP) tasks. In the former task, the annotator is presented with two sentences 
for each question. He needs to decide whether the second sentence can be inferred from the first. 
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Figure 8: The figure compares the estimator accuracy assuming that the marginal p(y) is misspec- 


ified. The plots draw mae(ĝ™le, o"e) as a function of n for k = 1 and 6™™!° = 0.75 when 
P™ (y = 1) = 0.8 (left) and P™° (y = 1) = 0.75 (right). Small perturbations in P™*(y) do 
not affect the results significantly; interestingly over-specifying P™® (y = 1) leads to more 
accurate estimates than under-specifying (misspecification closer to uniform distribution) 
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Figure 9: Mean prediction accuracy for the unsupervised predictor combination scheme in (4) for synthetic 


data. The left panel displays classification accuracy and the right panel displays the regression ac- 
curacy as measured by 1 — 1 gY- ywy. The graphs show that in both cases the accuracy 
increases with k and n in accordance with the theory and the risk estimation experiments. 


The original data set contains 800 sentence pairs with a total of 165 annotators. The latter task 
involves recognizing the temporal relation in verb-event pairs. The annotator is forced to decide 
whether the event described by the first verb occurs before or after the second. The original data 
set contains 462 pairs and 76 annotators. In both data sets, most of the annotators have completed 
only a handful of tasks. Therefore, we selected a subset of these annotators for each task such that 
each annotator has completed at least 100 problems and has differing accuracies. The data sets 
contain ground truth labels which are used solely to calculate the annotator accuracy and not used 
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Figure 10: mae(6™*,@'"*) as a function of n for different number of annotators k on RTE 
(left) and TEMP (right) data sets. Left: n = 100, P(y = 1) = 0.5 and 6" = 
{0.85,0.92,0.58,0.5,0.51}. Right: n = 190, P(y = 1) = 0.56 and o™ = 
{0.93,0.92,0.54,0.44,0.92}. The classifiers were added in the order specified. 


at all during the estimation process. For efficiency, we selected only the instances for which all 
annotators provide an answer. This resulted in n = 100,190 for RTE and TEMP, respectively. 

In Figure 10 we display mae(0™°, ĝmle) for these data sets as function of n for different values of 
k. These plots generated from real-world data show similar trend to the synthetic experiments. The 
estimation errors decay to 0 as n increases and generally tend to decrease as k increases. This corre- 
spondence is remarkable since two of the labelers have worse than random accuracy and since it is 
not clear whether the conditional independence assumption actually holds in reality for these data 
sets. Nevertheless, the collaborative estimation error behaves in accordance with the synthetic data 
experiments and the theory. This shows that the estimation framework is robust to the breakdown 
of the assumption that the classifier accuracy must be higher than random choice. Also, whether the 
conditional independence assumption holds or not is not crucial in this case. 

We further experimented with classifiers trained on different representations of the same data 
set and estimated their error rates. We adopted the Ringnorm data set generated by Breiman (1996). 
Ringnorm is a 2-class artificial data set with 20 dimensions where each class is drawn from a multi- 
variate normal distribution. One class has zero mean and a covariance & = 47 where / is the identity 
matrix. The other class has unit covariance and a mean u = ( Fo Fo ites Fag): The total size 
is 7400. We created 5 different representations of the data by projecting it onto mutually exclusive 
sets of principal components obtained by Principal Component Analysis (PCA). We trained an SVM 
classifier (with 2-degree polynomial kernel) (Vapnik, 2000; Joachims, 1999) on samples from each 
representation while holding out 1400 examples as the test set resulting in a total of 5 classifiers. We 
tested each of the 5 classifiers on the test set and used their outputs to estimate the corresponding 
parameters. The true labels of the test set examples were used as ground truth to calculate the mae 
of the mle estimators. 

The mae curves for this data set appear in Figure 11 as a function of the number n of unlabeled 
examples. When all classifiers are highly accurate (upper left panel), the collaborative unsupervised 
estimator is reliable, see Figure 11(a). With a mixture of weak and strong classifiers (upper right 
panel), the collaborative unsupervised estimator is also reliable. This is despite the fact that some of 
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Figure 11: mae(0""°,ĝ™le) as a function of the test set size on the Ringnorm data set. p(y = 1) = 
0.47, and 0*™"° is indicated in the legend in each plot. The four panels represent mostly 
strong classifiers (upper left), a mixture of strong and weak classifiers (upper right), 
mostly weak classifiers (bottom left), and mostly very weak classifiers (bottom right). 
The figure shows that the framework is robust to occasional deviations from the as- 
sumption regarding better than random guess classification accuracy (upper right panel). 
However, as most of the classifiers become weak or very weak, the collaborative unsu- 
pervised estimation framework results in worse estimation error. 


the weak classifiers in Figure 11(b) have worse than random accuracy which violates the assump- 
tions in the consistency proposition. This shows again that the estimation framework is robust to 
occasional deviations from the requirement concerning better than random classification accuracies. 
On the other hand, as most of the classifiers become worse (bottom row), the accuracy of the un- 
supervised estimator decreases, in accordance with the theory developed in Sections 5 (recall the 
Fisher information contour plot). 

Our experiments thus far assumed the symmetric noise model (8). Despite it not being always 
applicable for real world data and classifiers, it did result in good estimation accuracy in some of the 
cases described thus far. However, in some cases this assumption is grossly violated and the more 
general noise model is needed (6). For this reason, we conducted two experiments using real world 
data assuming the more general (6). 

The first experiment concerned domain adaptation (Blitzer et al., 2007) for Amazon’s product 
reviews in four different product domains: books, DVDs, electronics and kitchen appliances. Each 
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| book | dvd | kitchen | electronics || 20newsgroup 
| training error 0.22 | 0.23 | 0.26 0.30 0.028 

| non-collaborative | 0.04 | 0.04 | 0.08 0.06 0.006 

| collaborative 0.10 | 0.10 | 0.09 0.08 n/a 


























Figure 12: mae(6™*,@'"*) for the domain adaptation (n = 1000, p(y = 1) = 0.75) and 20 news- 
group (n = 15,000, p(y = 1) = 0.05 for each one-vs-all data). The unsupervised non- 
collaborative estimator outperforms the collaborative estimator due to violation of the 
conditional independence assumption. Both unsupervised estimators perform substan- 
tially better than the baseline training error rate estimator. In both cases the results were 
averaged over 50 random train test splits. 


domain consists of positive (y = 1) and negative (y = 2) reviews with p(y = 1) = 0.75. The task was 
to estimate the error rates of classifiers (linear SVM, Vapnik, 2000; Joachims, 1999) that are trained 
on 300 examples from one domain but tested on other domains. The mae values for the classification 
risks are displayed in Figure 12 with the columns indicating the test domain. In this case, the 
unsupervised non-collaborative estimator outperforms the collaborative estimator due to violation 
of the conditional independence assumption. Both unsupervised estimators perform substantially 
better than the baseline estimator that uses the training error on one domain to predict testing error 
on another domain. 


In the second experiment using (6) we estimated the risk (non-collaboratively) of 20 one vs. all 
classifiers (trained to predict one class) on the 20 newsgroup data (Lang, 1995). The train set size 
was 1000 and the unlabeled data size was 15000. In this case the unsupervised non-collaborative 
estimator returned extremely accurate risk estimators. As a comparison, the risk estimates obtained 
from the training error are four times larger than the unsupervised MLE estimator (See Figure 12). 


8. Discussion 


We have demonstrated a collaborative framework for the estimation of classification and regression 
error rates for k > 1 predictors. In contrast to previous supervised risk estimation methods such 
as cross validation (Duda et al., 2001), bootstrap (Efron and Tibshirani, 1997), and others (Hand, 
1986), our approach is fully unsupervised and thus able to use vast collections of unlabeled data. 
Other related work includes Smyth et al. (1995) and Sheng et al. (2008) which consider repeated 
labeling where each instance is labeled by multiple experts and the final label is decided based on a 
majority voting scheme. However, Smyth et al. and Sheng et al. fail to address estimating the risks 
of the predictors which is the main focus of our work. 


We prove statistical consistency in the unsupervised case and derive the asymptotic variance. 
Our experiments on synthetic data demonstrate the effectiveness of the framework and verify the 
theoretical results. Experiments on real world data show robustness to underlying assumptions. The 
framework may be applied to estimate additional quantities in an unsupervised manner, including 
noise level in noisy communication channels (Cover and Thomas, 2005) and error rates in structured 
prediction problems. 
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